Systems and methods for estimating embryo viability

ABSTRACT

A computer-implemented method, including the steps of: receiving video data of a human embryo, the video data representing a sequence of images of the human embryo in chronological order; applying at least one three-dimensional (3D) artificial neural network (ANN) to the video data to determine a viability score for the human embryo, wherein the viability score represents a likelihood that the human embryo will result in a viable embryo or a viable fetus; and outputting the viability score.

RELATED APPLICATIONS

This is an application related to Australian Provisional PatentApplication Nos. 2017905017 entitled “Systems and methods fordetermining embryo viability” filed on 15 Dec. 2017, and 2018901754entitled “Systems and methods for estimating embryo viability” filed on18 May 2018. These related applications are incorporated herein byreference and made a part of this application.

TECHNICAL FIELD

The present disclosure relates to systems and methods for estimatinghuman embryo viability.

BACKGROUND

In vitro fertilisation (IVF) refers to processes/techniques where awoman's eggs are fertilised outside of the body. IVF typically involvesadministering fertility drugs to women to stimulate the maturation ofmultiple follicles as opposed to a single follicle in the normal cycle.These eggs are then retrieved surgically and transported to a laboratorywhere they are fertilised with the male partner's sperm or donatedsperm. The fertilised eggs are then allowed to mature as embryos in aspecialised culture environment, typically within an incubator. Duringthis time, microscopic images of the developing embryos can be acquiredusing imaging technology within the incubator (such as an Embryoscope®incubator) to produce continuous time-lapse videos of the developingembryos.

Traditionally, multiple embryos are implanted to the woman's uterus toincrease the overall success rate. The disadvantage of this approach isthe increase in the probability of multiple pregnancies, which areassociated with a higher risk of antenatal complications. As a result,one goal for improvement of IVF is to be able to perform a single embryotransfer for each pregnancy. The term “transfer” refers to a step in theprocess of assisted reproduction in which embryos are placed into theuterus of a female with the intent to establish a pregnancy.

To achieve this, one must be able to select a single embryo out ofmultiple developed embryos in accordance with highest pregnancypotential. This selection process is currently performed byembryologists who manually grade each embryo based on its appearance andthe timing of critical developmental checkpoints.

Currently, the quality of each embryo is determined using a number ofgrading schemes. These schemes involve manual annotation of each embryoimage or time-lapse video. Features that are considered in these gradingsystems include the morphological appearance of embryo as well as theprecise timing of key developmental checkpoints. Currently, allsolutions are purely workflow tools for embryologists. They dependentirely on the subjective judgement of the embryologist annotating eachembryo. Some commonly used grading systems are the Gardner BlastocystGrading System (https://www.advancedfertilitv.com/blastocystimages.htm)and KIDScore(http://www.vitrolife.com/sv/Products/EmbryoScope-Time-Lapse-System/KIDScore-decision-support-tool-/).

However, the selection process is an inexact science and is highlyvariable depending on each embryologist. Embryologists are required tomake subjective judgements about the exact timing of certaindevelopmental checkpoints as well as symmetry, size and uniformity. Thisis highly dependent on the operator's experience and personal opinion.This means embryologists often disagree with other embryologists or eventhemselves (when shown the same embryo again) on which embryo has thehighest potential for transfer. As such, there is poor reproducibilityand high inter- and intra-reader variability amongst embryologists. Itis a time-consuming and labour-intensive process to label eachtime-lapse video. Manual embryo grading typically requires up to 1 hourper patient. It is unclear which features or which combination offeatures are ultimately predictive towards the pregnancy potential ofeach embryo. Current grading methods typically only analyse 2 to 4isolated time frames that have been shown to independently result in ahigher pregnancy rate. Furthermore, current systems, like theEmbryoscope®, allow variable selection/deselection of multipleannotation/analysis parameters which may discourage analysis of howthese aspects interact.

It is desired to address or ameliorate one or more disadvantages orlimitations associated with the prior art, or to at least provide auseful alternative.

SUMMARY

Provided herein is a computer-implemented method, including the stepsof:

receiving video data of a human embryo, the video data representing asequence of images of the human embryo in chronological order;

applying at least one three-dimensional (3D) artificial neural network(ANN) to the video data to determine a viability score for the humanembryo, wherein the viability score represents a likelihood that thehuman embryo will result in a viable embryo or a viable fetus; and

outputting the viability score.

Provided herein is a system, including at least one processor configuredto:

receive video data of a human embryo, the video data including asequence of images of the human embryo in chronological order;

apply at least one three-dimensional (3D) artificial neural network tothe video data to determine a viability score for the human embryo,wherein the viability score represents a likelihood that the humanembryo will result in a viable embryo or a viable fetus; and

output the viability score.

Provided herein is a method including:

generating a viability score for a human embryo by probabilisticallyclassifying data representing a video of the human embryo with anartificial neural network (ANN); or

training an artificial neural network (ANN) to probabilisticallyclassify data representing a video of a human embryo to generate aviability score for the human embryo.

Provided herein is a system, including at least one processor configuredto:

generate a viability score for a human embryo by probabilisticallyclassifying data representing a video of the human embryo with anartificial neural network (ANN); or

train an artificial neural network (ANN) to probabilistically classifydata representing a video of a human embryo to generate a viabilityscore for the human embryo.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present invention are hereinafter described, byway of example only, with reference to the accompanying drawings,wherein:

FIG. 1 shows a general structure of an exemplary system for estimatingembryo viability;

FIG. 2 shows an exemplary architecture of a 3D convolutional neuralnetwork used in one example;

FIGS. 3A and 3B show an exemplary architecture of a 3D convolutionalneural network schematic used in another example;

FIG. 4 shows one example of a heat map produced by the system;

FIG. 5A shows an example of time-lapse video data;

FIG. 5B illustrates application of a 3D occlusion window and heat mapsgenerated for the time-lapse video data;

FIG. 6 shows method/processing steps for generating a heat map;

FIG. 7 shows method/processing steps for estimating embryo viability;

FIG. 8 shows an example of a computer for training the deep learningmodel;

FIG. 9 illustrates ensembling of multiple neural networks according toone example;

FIG. 10 is a graph showing performance of the deep learning modelaccording to one example;

FIG. 11 is a graph showing an exemplary correlation between outputviability score and actual pregnancy outcome;

FIG. 12 is one example of a software interface of the system;

FIG. 13 is a schematic overview of one example ofimplementation/training of a classification model for thesystems/methods;

FIG. 14 shows another example of the heat map;

FIG. 15 shows the labelling of the time-lapse videos used for trainingthe deep learning model;

FIG. 16 shows the split of a full dataset into a training set and atesting set;

FIGS. 17A and 17B show an example of images at slightly different pointsin time in a time-lapse embryo video before and after pre-processing,respectively;

FIG. 18 shows a division of the dataset when ensembling multiple neuralnetworks; and

FIG. 19 shows an exemplary user interface of the software package thatdisplays the viability score.

DETAILED DESCRIPTION

Embodiments of the present invention provide a processing system forestimating embryo viability. The system is configured to receive videodata of a human embryo and process the received video data to determinea viability score for the embryo. The video data includes a sequence ofimages in chronological order, so it is also referred to as “time-lapsevideo data”.

It will be appreciated that the term “embryo” is intended to include thezygote or fertilized ovum, as well as the embryo that developstherefrom.

Generally, the viability score is or includes a probability, providing aprediction of the likelihood of an embryo leading to a successfulpregnancy after implantation in the uterus. The embryo with a higherscore has a higher probability of resulting in a viable embryo or aviable human fetus.

The viability scores may be used for determining, among multiple embryosincubated for a patient, a single embryo to be transferred into theuterus of a female. For example, the embryo with a higher score may beselected to be implanted in the uterus. This may prevent the risk ofantenatal complications associated with multiple pregnancies due totransferring multiple embryos. Determining the embryo with the highestprobability of resulting in a viable embryo or a viable fetus alsodecreases the time to pregnancy as the best embryo is transferred first,avoiding a failed transfer that necessitates a subsequent embryotransfer.

Alternatively, when multiple embryos are to be transferred (duringsubsequent treatment cycle using frozen embryos), the viability scoremay be used to decide the order in which embryos will be transferredinto the uterus of a patient.

Some existing embryo assessment methods or grading systems (e.g.,Gardner Blastocyst Grading System and KIDScore) may classify an embryointo a limited number of grades, e.g., a grade between 1 to 5.

By contrast, the present disclosure provides classification with a finergrain by estimating the probability of the embryo resulting in a viableembryo or a viable fetus. The classification methods used in the presentdisclosure may also be referred to as “probabilistically classifying”methods. This probabilistic classification provides a probability value,e.g., a percentage for each embryo, thus identifying finer differencesbetween embryos. Accordingly, even embryos with the same grade accordingto existing grading systems can be ranked by the viability score. Thisallows for automatic generation of a ranking of a plurality of embryosbased on their viability, and automatic selection of a single embryo fortransfer from the plurality of embryos based on the ranking.

A viable embryo may be defined as an embryo having:

a biochemical pregnancy detected based on a urine test or a blood test(e.g., for 3-HCG); or

a clinical pregnancy with a viable gestational sac or a viable yolk sacdetected on ultrasound at a predetermined time (e.g., between 6-7 weeks)after embryo transfer.

A viable fetus may be defined as having:

a viable fetal heart detected on maternal ultrasound a selected time(e.g., 6 or more weeks) following the embryo transfer; or

a live birth at the end of the pregnancy.

Compared to some known measurements of embryo quality, e.g., embryograde determined subjectively by an embryologist using existing gradingschemes based on “how good the embryo looks”, or implantation potentialthat represents the likelihood that the mother will have a positivepregnancy test following the embryo transfer, using the ultrasound fetalheart detection result 6 weeks following the embryo transfer provides amore objective and more reliable estimation of the viability of theembryo.

It will also be appreciated that the term “processing system” may referto any electronic processing device or system, or computing device orsystem, or combination thereof (e.g., computers, web servers, smartphones, laptops, microcontrollers, etc.). The processing system may alsobe a distributed system. In general, processing/computing systems mayinclude one more processors (e.g., CPUs, GPUs), memory componentry, andan input/output interface connected by at least one bus. They mayfurther include input/output devices (e.g., keyboard, displays etc.). Itwill also be appreciated that processing/computing systems are typicallyconfigured to execute instructions and process data stored in memory(i.e., are programmable via software to perform operations on data).

FIG. 1 illustrates a general structure of an exemplary system 100 forestimating embryo viability. The system 100 is an example of theprocessing system.

As shown, the system 100 includes an incubator 102 for containing anembryo 104 and maintaining environmental conditions suitable for theembryo 104 to live. The incubator 102 includes an image sensor 106 forcapturing time-lapse video data of the embryo 104.

The time-lapse video data captured by the image sensor 106 is sent to aprocessor 108 which applies a deep learning model to the time-lapsevideo data to determine a viability score for the embryo 104.

The viability score determined by the processor 108 may be subsequentlyoutput to a display 110 or other suitable output device for use by humanstaff, such as an embryologist.

It will be appreciated that no manual feature extraction or humanannotation of the video data is required, and that the deep learningmodel is an end-to-end model receiving nothing but the raw video data tooutput the viability score.

In order to determine the viability score, the deep learning model isapplied to the time-lapse video data. The deep learning module includesat least a three-dimensional (3D) artificial neural network (ANN), suchas a 3D convolutional neural network (3D CNN).

A 3D CNN extracts features from both the spatial and the temporaldimensions by performing 3D convolutions, thereby capturing not onlyinformation contained in each single image frame in the video but alsothe motion information contained in multiple chronologically separatedimage frames, including adjacent image frames.

This is in contrast to analysing embryo quality by applying a machinelearning model to only static images of the embryo, which takes intoaccount only the information contained in each static image.

This is also in contrast to systems where humans are required to extractfeatures such as morphological grading manually, or to annotate theexact timing of developmental milestones. Such systems may apply machinelearning, but only to these extracted features and/or annotations, andto predict only the embryo grading. Accordingly, analysis of embryotime-lapse video using such systems may rely on the experience of theembryologist who manually or semi-automatically annotates features(e.g., morphological grading) or extracts timing of key developmentalmilestones. This process may be time-consuming and inaccurate. Forexample, each patient may have up to 30-20 embryos per treatment cycle,and each embryo may take up to 5 minutes to fully annotate. Accordingly,this is not a scalable solution for analysing a large number oftime-lapse videos of embryos. By contrast, a fully end-to-end method foranalysing time-lapse embryo videos using a 3D ANN may analyse, e.g., 10embryos per second on a typical laptop, which is more efficient than theexisting methods, and thus can make time-lapse video analysis of embryosscalable.

The system described herein extracts not only intra-frame features butalso inter-frame features of the time-lapse video data, thereforecapturing both the spatial and temporal features of the embryo. In thisway, the described system may provide more comprehensive and moreaccurate analysis of the viability of an embryo compared to the existingmethods.

FIG. 2 illustrates an exemplary architecture of the 3D CNN. In thisexample, the 3D CNN contains a series of convolution and pooling layers.More specifically, the CNN includes the repeated application of two3×3×3 convolutions followed by a pooling operation. The last layer ofthe 3D CNN is a prediction layer which outputs the viability score.

FIGS. 3A and 3B illustrate another exemplary architecture of the 3D CNN.In this example, the 3D CNN is an existing inflated 3D CNN, generated byinflating the two-dimensional (2D) filters and pooling kernels of a 2DCNN into 3D, as described in J Carreira, A Zisserman (2017), Quo vadis,action recognition? a new model and the kinetics dataset, 2017 IEEEConference on Computer Vision and Pattern Recognition (CVPR), 4724-4733.Accordingly, an N×N filter becomes an N×N×N filter after the inflation.As shown in FIG. 3A, the inflated 3D CNN has a plurality of convolutionand polling layers, and a plurality of inception modules (“Inc.”). FIG.3B illustrates the architecture of the inception module.

The last layer of the 3D CNN shown in FIG. 3A is a linear classificationlayer, which outputs the viability score of the embryo.

As shown in FIG. 2 and FIGS. 3A and 3B, both exemplary architectures ofthe 3D CNN utilises 3D convolution kernels and pooling kernels, whichallow the 3D CNN to capture spatio-temporal features of the embryo fromthe time-lapse video data.

It will be appreciated that the video data of the embryo may be derivedfrom variety or formats, such as, for example, a sequence of stillimages in chronological order; or a time-lapse video document. In oneexample, the time-lapse video data is a time-lapse video documentincluding 720 time-lapse image frames.

The 3D CNN is trained by using:

video data representing a plurality of sequences of images of aplurality of human embryos; and

pregnancy outcome data that indicates whether each of the plurality ofhuman embryos has resulted in a viable embryo or a viable foetus.

As described hereinbefore, a viable embryo may be defined as an embryohaving:

a biochemical pregnancy detected based on a urine test or a blood test(e.g., for 3-HCG); or

a clinical pregnancy with a viable gestational sac or a viable yolk sacdetected on ultrasound at a predetermined time (e.g., between 6-7 weeks)after embryo transfer.

A viable fetus may be defined as having:

a viable fetal heart detected on maternal ultrasound a selected time(e.g., 6 weeks, 7 weeks, 8 weeks, 9 weeks, 10 weeks, 11 weeks, 12 weeks,3 months, 4 months, 5 months, 6 months, 8 months, 9 months) followingthe embryo transfer; or

a live birth at the end of the pregnancy.

Some existing machine-learning-based embryo assessment methods mayrequire pre-analysis to manually determine which features of the embryo(e.g., blastomere symmetry, cytoplasmic appearance, and/or amount offragmentation) to extract and analyse, or human annotation of key events(e.g., neural neurogenesis, musculoskeletal somitogenesis, and/or heartcardiogenesis) in the development of the embryo. By contrast, the 3D CNNdescribed herein can be trained and used without manual selection orextraction of embryo characteristics/features, or human annotation ofkey development events. In other words, the deep learning modeldescribed herein provides an end-to-end embryo assessing process. Thismay be advantageous because medical professionals currently do not havea complete understanding in relation to what characteristics/featuresare the most useful characteristics/features in assessing the quality ofa human embryo. Therefore, by providing an end-to-end embryo assessmentprocess, the deep learning model described herein allows the system tolearn and automatically determine which characteristics/features shouldbe used, and thus can provide more accurate results than existing embryoassessment methods and systems. Furthermore pre-analysis to extractfeatures such as symmetry, number of cells, degree of fragmentation andexact timing of key events is a subjective and non-repeatable process,and is highly variable between embryologists. A deep learning modelapplied to these data would be subjected to the same bottleneck in termsof performance.

The 3D CNN may be trained on one or more devices different from theprocessor 108. For example, 3D CNN may be trained by using a deviceincluding one or more graphical processing units (GPUs) and/or one ormore central processing units (CPU).

Referring back to FIG. 1, in some embodiments, the processor 108 maypre-process the time-lapse video data before applying or training thedeep learning module.

For example, the processor 108 may standardise the received time-lapsevideo data so that all videos span a predetermined time period.

The processor 108 may also perform a cropping step to retainpredetermined areas in the time-lapse video data, e.g., areas thatincludes the embryo.

The processor 108 may further adjust contrast of the images in thetime-lapse video data to enhance the video quality, for example byapplying a contrast limited adaptive histogram equalisation (CLAHE).

Finally, the processor may resize the images in the time-lapse videodata to a predetermined image size.

In some forms, the system 100 is further configured to produce a visualoverlay for display over at least some images of the time-lapse videodata. The visual overlay is indicative of the contribution of parts ofthe images to the viability score.

In one example, the visual overlay is a heat map (also referred to as acontribution map), an example of which is shown in FIG. 4.

The heat map may be generated by analysing change of the viability scoreoutput by the deep learning model when occluding portions of the imagesin the time-lapse video data. For example, a 3D occlusion window can beapplied to the video to occlude different portions of the video.

FIG. 5A shows an example of the time-lapse video data before adding theheat map. FIG. 5B illustrates the application of a 3D occlusion windowand the generated heat map.

As shown in FIG. 5B, different intensities or colours may be used torepresent levels of contribution toward embryo viability of eachspatio-temporal region. For example, blue may indicate spatio-temporalregions that have low levels of contribution to the embryo viability(such a region may also be referred to as an “unfavourablespatio-temporal region”), while red may indicate spatio-temporal regionsthat have high levels of contribution to the embryo viability (such aregion may also be referred to as a “favourable spatio-temporalregion”).

FIG. 6 illustrates an exemplary processing flow performed by theprocessor 108 in generating the heat map.

At Step 610, the processor 108 uses the 3D CNN model to predict anoriginal viability score based on the time-lapse video data.

At Step 620, the processor 108 applies a 3D occlusion window to acorresponding 3D portion of the video data to occlude a 3D region of thevideo data with black pixel.

At Step 630, the processor 108 uses the same 3D CNN model to predict anew viability score based on the partially occluded time-lapse video.

At Step 640, the processor 108 determines a difference between the newviability score and the original viability score for the currentposition of the 3D occlusion window.

At Step 650, the processor 108 determines whether the whole video hasbeen processed.

If not, the processor 108 moves the 3D occlusion window to the next 3Dregion at Step 660 and then loops back to Step 620.

If the whole video has been processed, the processor 108 proceeds toStep 670 to generate a 3D heat map that represents the degree ofcontribution of each spatio-temporal region toward the embryo viability.

Finally, at Step 680, the processor 108 outputs the generated 3D heatmap.

The generated 3D heat map may be subsequently sent to a display device,such as the display 110, where the 3D heat map may be presented to humanstaff, e.g., an embryologist. This allows an embryologist to reviewand/or analyse the decision-making process of the machine learningmodel, and to learn the characteristics/features in the embryo videoused by the machine learning model to assess the viability of theembryo. In this way, the heat map may improve the understanding of humanmedical professionals in embryo viability and help in diagnosing anyabnormal behaviour by the system.

It will also be appreciated that the processor 108 as described hereinmay be integrated into other apparatuses and systems, such as, forexample the incubator 102 used for storing and/or developing embryosprior to implantation. Accordingly, an incubator can incorporate aprocessing system, such as the processor 108 described herein.Alternatively, the processor 108 may be provided as an apparatusseparate from the incubator 102, as shown in FIG. 1, e.g., in cloudservers.

Further, although in the example shown in FIG. 1 the processor 108 is indata communication with the incubator 102 and receives the time-lapsevideo data captured by the image sensor 106 directly from the incubator102, in some embodiments, the time-lapse video data may be stored on adata store or a server which can be accessed or read by the processor108 subsequently.

In some embodiments, the processor 108 may be communicably connected tothe incubator 102 or the data store that stores the time-lapse videodata via one or more wired and/or wireless networks. The determinationof the viability of the embryo may be provided as web/cloud basedservice/application i.e., to be accessed via the Internet.

It will be appreciated that in addition to being embodied as astand-alone system, or incorporated as part of another apparatus orsystem (e.g., incubator), embodiments of the present invention mayinclude a method to be performed by a computer (or other suitableelectronic processing device).

In such forms, embodiments provide a computer implemented method ofestimating viability of human embryo for implantation. As shown in FIG.7, the method includes the steps of:

receiving video data of a human embryo, the video data including asequence of images of the human embryo in chronological order (Step710);

applying at least one three-dimensional (3D) artificial neural networkto the video data to determine a viability score for the human embryo,wherein the viability score represents a likelihood that the humanembryo will generate an ongoing pregnancy (Step 720); and

outputting the viability score (Step 730).

In some embodiments, the method may further include:

determining a ranking of a plurality of human embryos based on theirviability scores, as shown by Step 740 in FIG. 7.

In some embodiments, the method may further include:

selecting, based on the ranking, one of the plurality of human embryosfor a single embryo transfer or the order in which multiple embryosshould be transferred, as shown by Step 750 in FIG. 7.

The selected embryos may be subsequently transferred into the uterus ofa patient.

Also provided herein is a computer program including instructions thatconfigure a computer to perform the method as described herein, whichmay be provided on a computer readable medium. In one example, themethod is implemented on a remote server (e.g., cloud based server) tobe accessed via a communication network (e.g., the internet).

Also provided herein is a method including:

generating a viability score for a human embryo by probabilisticallyclassifying data representing a video of the human embryo with anartificial neural network (ANN); or

training an artificial neural network (ANN) to probabilisticallyclassify data representing a video of a human embryo to generate aviability score for the human embryo.

Further provided herein is a system, including at least one processorconfigured to:

generate a viability score for a human embryo by probabilisticallyclassifying data representing a video of the human embryo with anartificial neural network (ANN); or

train an artificial neural network (ANN) to probabilistically classifydata representing a video of a human embryo to generate a viabilityscore for the human embryo.

The presently described systems and methods may provide severaladvantages over conventional methods for estimating/predicting theviability of embryos.

For example, in implementing the system/method, human error may bereduced/removed the process of assessing the embryo quality. The systemis objective and is not influenced by fatigue, emotional bias orinexperience. The viability score provided to each embryo is alsoreproducible and there is no variability between readers or labs.

The training of the deep learning model described herein do not requiremanual human labelling/annotation of embryo characteristics/features.The system/method described herein provides an end-to-end embryoassessment solution. As described hereinbefore, given that medicalprofessionals currently do not have a comprehensive understanding of thecharacteristics/features suitable for assessing embryo quality, anend-to-end process can provide more accurate results thansystems/methods that relying on manual selection/annotation of embryocharacteristics/features. Furthermore the annotation and featureextraction step is very labour intensive typically take 5 to 10 minutesper embryo. Each treatment cycle can have up to 50 embryos.

The system/method may interpret time-lapse video much faster than ahuman embryologist. When implemented on a typical personal computer, thesystem/method can interpret about 10 embryos per 1 second. It is thushighly scalable for mass adoption. In some examples, the speed may besuch to allow embryos to be interpreted almost instantly on demand,making patient scheduling more flexible.

The operational cost of implementing the system/method may be muchcheaper than that of a highly trained embryologist. As a result, IVFlaboratories can allocate their highly paid human resources toward otheraspects of IVF.

In addition, the visual overlay, such as the heat map generated usingthe occluding window, allows embryologists to learn from the machinelearning model. By using the heat map, the system/method describedherein can empower embryologists and can be used as a tool foridentifying unknown markers for embryo viability.

Example 1

A software tool/application for predicting pregnancy potential/viabilityof embryos by analysing time-lapse videos from incubators (e.g.,Embryoscope®) was developed for implementation on a processing/computersystem. The software implemented a deep learning model with 3D CNNnetworks.

A training dataset of embryo time lapse videos with known pregnancyoutcomes was used to train the 3D CNN deep learning model. The trainingdataset included 903 time lapse videos, 657 with negative pregnancyoutcomes and 246 with positive pregnancy outcomes. The videos wererandomised into a training set (75%) and a testing set (25%) for posttraining validation.

The processing/computer system included a personal computer with fourgraphical processing units (GPUs) and 12 core central processing units(CPU), as shown in FIG. 8. The system was cooled by two separate waterloops along with 12 high performance fans to prevent overheating.

As shown in FIG. 9, the 3D CNN deep learning model used in this exampleincluded three separate neural networks. To determine the viabilityscore, each embryo time-lapse video was independently analysed by eachof the three neural networks and the average of their predictions wasreported as final output. The model returned a probability (viability)score that a given embryo will go on to create a clinical pregnancy(0%-100%).

After the training process, the model scored 180 of the embryos from thetesting set. As shown in FIG. 10, the deep learning model achieved anarea under the curve (AUC) of 0.78, out-performing human graders(AUC=0.55-0.75). At a cut-off threshold of 20%, the model had asensitivity of 95%, specificity of 43%, positive likelihood ratio of1.65 and negative likelihood ratio of 0.13.

The returned viability score was also well correlated to the actualpregnancy rate. FIG. 11 shows the pregnancy rate of embryos whencompared to viability score produced for those embryos. It is shown thatthe model could clearly discriminate embryos with high pregnancypotential vs. low pregnancy potential. In addition, its predictedprobability of pregnancy was very similar to the actual pregnancy rate.

The software tool allowed a user to overlay a heat map (as shown in FIG.4) that highlighted the point in time (i.e., during embryo development)as well as the location in space (i.e., part of video image) thatcontributed the most to the prediction provided by the model. The heatmap provided an insight to embryologists about the model's decisionmaking process. This heat map along with the viability score could be apart of the patient's treatment record.

In this particular example, the input format was any time-lapse video(e.g., .avi file) exported from the EmbryoScope® software including ICSIor IVF embryos. It will be appreciated that the videos could havedifferent starting times in the embryo development cycle (e.g., D5-D6Blastocyst stage, D2-3 Cleavage stage).

Multiple embryos/videos may be assessed simultaneously. In suchinstances, the software/system ranked the embryos in accordance withtheir viability score and the embryo with the highest score will berecommended for a single embryo transfer. In the example shown FIG. 12,“Embryo 1” was the recommended embryo. The remainder of the embryoscould then be frozen, and subsequently transferred in descending orderof viability score, so as to give the patient the highest chance of asuccessful pregnancy outcome. Furthermore, a specific cut-off thresholdcould be decided by each IVF clinic, and any embryo scoring less thanthis threshold could be deemed as “non-viable” and not be considered forfreezing or embryo transfer.

In this example, the software tool/application ran on the Linuxoperating system. It will be appreciated that other versions may readilybe produced to operate on different operating systems. It will beappreciated that the software/system may be deployed as web/cloud basedservice/application, i.e., to be accessed via the Internet. It will beappreciated that the model may be improved by adding more data into thetraining process. FIG. 13 shows an overview of one example of training,implementation and retraining of the deep learning model.

Example 2

A software tool/application for predicting pregnancy potential/viabilityof embryos by analysing time-lapse videos from incubators (e.g.,Embryoscope® or EmbryoScope+®) was developed for implementation on aprocessing/computer system. The software implemented a deep learningmodel included a 3D CNN as shown in FIGS. 3A and 3B.

Data Collection

Time-lapse videos of embryo exported from commercially availabletime-lapse incubators such as EmbryoScope® or EmbryoScope+® wascollected from IVF laboratory and used to train the deep learning model.

The outcome of each embryo was obtained from a patient management systemand was used to label these time-lapse videos using the schematic shownin FIG. 15.

In particular, embryos that were transferred to the patient and resultedin a fetal heart detectable on antenatal ultrasound at 6 weekspost-embryo transfer were labelled “1” for positive. Embryos that wereeither discarded by embryologist or did not result in a fetal heart werelabelled “0” for negative. All embryos with unknown or undeterminedoutcome were not used for training.

Data Splitting

Time-lapse videos of embryo exported from commercially availabletime-lapse incubators such as EmbryoScope® or EmbryoScope+® werecollected from IVF laboratory and used to train the deep learning model.

In total, the full dataset included 1281 time lapse videos.

As shown in FIG. 16, the full dataset was randomly split into a trainingset (80%) and a testing set (20%) The neural network was trained on thetraining set only. Once the training was completed, the model was testedon the testing set to obtain performance characteristic metrics.

The training set included 1025 time lapse videos, 789 with negativepregnancy outcomes and 236 with positive pregnancy outcomes. The testingset included 256 time lapse videos, 197 with negative pregnancy outcomesand 59 with positive pregnancy outcomes.

Data Preparation for Training

The time-lapse videos in the training dataset were pre-processed beforebeing used for training the deep learning model. Firstly, the time-lapsevideos were standardised in time to ensure that all embryo videos span 5days. A circular cropping function was then applied to each video tocentre the embryo and to block out unwanted areas, enabling the neuralnetwork to focus its learning toward the embryo. Contrast limitedadaptive histogram equalisation (CLAHE) was then applied to all imagesin the embryos videos to enhance the image quality. Finally, all embryovideos were resized to a fixed shape of 128×128×128×1 (128 frames of128×128 pixels and 1 channel of black and white).

FIGS. 17A and 17B show an example of images at slightly different pointsin time in a time-lapse embryo video before and after thepre-processing, respectively.

Data Augmentation

In order to increase the size of the original dataset, varioustransformations were randomly applied to each time-lapse video to createnew videos that were visually different to the original video. Thisallows the neural network to generalise better to unseen examples,thereby further improving the performance of the neural network.

These augmentation methods included:

Random 360-degree rotation;

Random horizontal and vertical flip;

Random re-scaling of the video size (1.2×-0.8×);

Random playback speed adjustment (1.2×-0.8×);

Gaussian blur to mimic out of focus effect; and

Randomly blacking out portions of the video.

During each step of the training process, a batch of video was randomlyselected from the dataset and the random set of augmentation operationswere applied to this batch to create slightly different set of videosfor training. This process was repeated as the entire dataset was loopedover multiple times.

Training of the Deep Learning Model

A 3D CNN deep learning model as shown in FIGS. 3A and 3B was used inthis example.

The 3D CNN was trained with the time-lapse video dataset usingstochastic gradient decent method. The loss function used for trainingwas categorical cross-entropy. The CNN was trained using a learning rateof 0.00001 and momentum of 0.9 for 102,400 steps. The learning rate wasthen lowered to 0.000001 and the network was trained for a further102,400 steps.

The training was performed using a personal computer with four graphicalprocessing units (GPUs), such as NVIDIA 1080 Ti, 6 core centralprocessing units (CPUs), such as Intel i7-6850K, and a 64 Gb RAM.

Ensembling Models

In this example, multiple 3D CNNs were ensembled to further improve theperformance of the CNN.

The ensembling method of choice was 5-fold cross-validation and modelbagging. As shown in FIG. 18, the full dataset was divided into fiveequal parts. One of these parts was sequentially held out for testingand the model was trained using the other 4 parts. The end results were5 unique neural networks that were trained on slightly differentdatasets and carried different biases. When making a prediction on a newembryo, all 5 neural networks were used to score the embryoindependently, and their scores were averaged to obtain the final score.This resulted in a unifying model that was more robust than anyindividual model alone.

Deployment Software

A software package was developed to apply the neural network to newtime-lapse videos. The software was configured to accept a video file ofthe time-lapse sequence and output a viability score (which may alsoreferred to as an “EB.Score”) for each embryo. The viability scorerepresented the likelihood that the given embryo would lead to a fetalheart on ultrasound 6 weeks after that embryo was transferred.

The human embryologist could subsequently make a decision on the bestembryo to be transferred based on the viability score.

FIG. 19 illustrates an exemplary user interface of the software packagethat displays the viability score (in the form of a percentage)generated for a number of embryos. The software package may be deployedon a suitable computer device, such as a laptop used in IVF labs. Asshown, the green bar (or light grey) represents the embryo with thehighest viability score (38.92%), the blue bars (or dark grey) representembryos that have acceptable viability scores, and the white barsrepresent embryos that have low viability scores (i.e., nonviableembryos).

Alternatively, the method may be implemented by any other suitablecomputing device, or provided as a web-based or cloud-based service thatcan be accessed via a network such as the Internet.

It will also be appreciated that the deep learning model may be improvedby adding more data into the training process. In other words, the deeplearning model is a self-improving and self-tuning model. Theperformance and robustness of the model can be further improved overtime by retraining the CNN, while more embryos are accessed by the deeplearning model.

Generating the Heat Map

In order to provide a visual representation of which areas of the videoresulted in a significant change in the returned viability score, heatmaps were generated by sequentially occluding parts of the video andrepeating the scoring process.

A heat map generated for an embryo video indicated areas of the videowhere the CNN was paying close attention to in making its finaldecision.

FIG. 14 shows examples of the heat map generated, in which blue regions(in the central area of the fourth and fifth image from the left)represent regions that have low levels of contribution to the embryoviability (such a region may also be referred to as an “unfavourableregion”), while red regions (in the central cell area of the sixth andseventh images from the left) represent regions that have high levels ofcontribution to the embryo viability (such a region may also be referredto as a “favourable region”).

The heat map allowed the neural network to communicate its decisionmaking process in a humanly readable way, thus improves thecollaboration between the neural network and human embryologists.

Interpretation

The reference in this specification to any prior publication (orinformation derived from it), or to any matter which is known, is not,and should not be taken as an acknowledgment or admission or any form ofsuggestion that that prior publication (or information derived from it)or known matter forms part of the common general knowledge in the fieldof endeavour to which this specification relates.

Many modifications will be apparent to those skilled in the art withoutdeparting from the scope of the present invention.

1. A computer-implemented method, including the steps of: receivingvideo data of a human embryo, the video data representing a sequence ofimages of the human embryo in chronological order; applying at least onethree-dimensional (3D) artificial neural network (ANN) to the video datato determine a viability score for the human embryo, wherein theviability score represents a likelihood that the human embryo willresult in a viable embryo or a viable fetus; and outputting theviability score.
 2. The method of claim 1, wherein the 3D artificialneural network includes a 3D convolutional neural network (CNN).
 3. Themethod of claim 1, wherein the viability score represents a probabilitythat transferring the human embryo will result in a viable fetus.
 4. Themethod of claim 1, wherein the viability score represents a probabilitythat transferring the human embryo will result in any one or more of thefollowing: a viable fetal heart detected within a predetermined periodof time after embryo transfer; a biochemical pregnancy detected based ona urine test or a blood test; a viable gestational sac or a viable yolksac detected on ultrasound at a predetermined time after embryotransfer; and a live birth at the end of pregnancy;
 5. The method ofclaim 1, wherein the 3D ANN is trained by using: video data representinga plurality of sequences of images of a plurality of human embryos; andpregnancy outcome data that indicates whether each of the plurality ofhuman embryos has resulted in a viable embryo or a viable fetus.
 6. Themethod of claim 5, wherein the training of the 3D ANN is performedwithout manual selection or extraction of features, or human annotationof key development events.
 7. The method of claim 2, wherein the 3D CNNincludes a plurality of convolution layers each using a 3D convolutionkernel, and a plurality of pooling layers each using a 3D poolingkernel.
 8. The method of claim 1, further including one or more of:standardising the received video data so that all videos span apredetermined time period; cropping the video data to retainpredetermined areas of the data; adjusting contrast of the images in thevideo data; and resizing the images in the video data to a predeterminedimage size.
 9. The method of claim 1, further including: processing thevideo data by adding a visual overlay to at least some images of thevideo data, the visual overlay indicative of contributions of respectiveportions of the images to the viability score; and outputting the imageswith the visual overlays.
 10. The method of claim 9, wherein the visualoverlay is a heat map.
 11. The method of claim 9, wherein the visualoverlay is generated by: determining changes of the viability scoreoutput caused by occluding portions of the images or sequence of imagesin the video data.
 12. The method of claim 11, wherein occludingportions of the images in the video data includes applying athree-dimensional occlusion window to the video data.
 13. The method ofclaim 1, further including: determining a ranking of a plurality ofhuman embryos based on their viability scores.
 14. The method of claim13, further including: selecting, based on the ranking, one of theplurality of human embryos for a single embryo transfer or the order inwhich multiple embryos should be transferred.
 15. A system, including atleast one processor configured to: receive video data of a human embryo,the video data including a sequence of images of the human embryo inchronological order; apply at least one three-dimensional (3D)artificial neural network to the video data to determine a viabilityscore for the human embryo, wherein the viability score represents alikelihood that the human embryo will result in a viable embryo or aviable fetus; and output the viability score. 16.-28. (canceled)
 29. Thesystem of claim 15, wherein the system includes: an image sensor forcapturing video data of the human embryo; wherein the processor isconfigured to receive the captured video data from the image sensor. 30.The system of claim 15, wherein the system includes a time-lapseincubator.
 31. A method including: generating a viability score for ahuman embryo by probabilistically classifying data representing a videoof the human embryo with an artificial neural network (ANN); or trainingan artificial neural network (ANN) to probabilistically classify datarepresenting a video of a human embryo to generate a viability score forthe human embryo.
 32. The method of claim 31, wherein the viabilityscore represents a likelihood that the human embryo will result in aviable embryo or a viable foetus.
 33. The method of claim 31, whereinthe video is of the human embryo in an incubator optionally for apreselected period of time; and/or wherein the video includes a seriesof images optionally including time-lapse images.
 34. The method ofclaim 31, wherein the ANN is a three-dimensional (3D) ANN, and/or aconvolutional neural network (CNN).
 35. The method of claim 31, whereinthe ANN is trained using: data representing a plurality of videos ofhuman embryos; and data representing whether the human embryos haveresulted in respective viable embryos or viable fetuses.
 36. A system,including at least one processor configured to: generate a viabilityscore for a human embryo by probabilistically classifying datarepresenting a video of the human embryo with an artificial neuralnetwork (ANN); or train an artificial neural network (ANN) toprobabilistically classify data representing a video of a human embryoto generate a viability score for the human embryo. 37.-40. (canceled)